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FOREWORD 


In  the  early  nineties,  the  proposition  that  advanced  distributed  simulation  (ADS)  was  the  wave 
of  the  future  for  test  and  evaluation  (T&E)  was  advanced.  Reaction  was  mixed.  At  one  end  of 
the  spectrum  were  people  who  believed  the  need  for  live  testing  would  fade  away,  and  at  the 
other  end  were  people  who  scoffed  at  the  notion  that  ADS  had  any  utility  at  all  for  testers.  At 
the  policy-making  level,  expectations  were  high  and  skepticism  was  subdued.  At  the 
implementation  level,  expectations  were  low  and  skepticism  was  high. 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E) 
program  was  chartered  in  October  1994  to  conduct  an  objective  assessment  of  the  worth  of 
ADS  for  support  of  T&E.  The  joint  test  force  (JTF)  conducted  tests  in  three  functional  areas: 
precision  guided  munitions  (PGMs);  command,  control,  communications,  computers, 
intelligence,  surveillance  and  reconnaissance  (C4ISR);  and  electronic  warfare  (EW).  While 
the  JTF  effort  was  resource  constrained,  its  results  have  broad  relevance.  Each  of  the  test 
areas  is  documented  in  executive-level  utility  reports.  This  report  addresses  the  utility  of  ADS 
in  EW  testing. 

This  report  suggests  that  there  is  a  range  of  utility  for  ADS  in  EW  testing.  The  greatest  utility 
is  expected  in  systems  of  systems  testing  such  as  testing  electronic  support  systems.  These 
systems  consist  of  multiple  C4ISR-based  EW  platforms,  rely  on  information  systems 
technology,  and  are  relatively  immune  to  the  effects  of  latency  and  data  loss  that  are  inherent 
in  ADS  architectures.  Integrated  EW  systems  are  expected  to  benefit  from  ADS  as  well, 
especially  where  a  single  facility  is  unable  to  test  all  the  EW  functions  simultaneously  in  a 
single  test  event.  Transporting  the  system  integration  laboratory  from  the  contractor’s  facility 
to  a  government  test  facility  is  impractical  for  highly  integrated  EW  systems.  Federated 
systems  or  single  function  EW  systems  are  generally  adequately  tested  using  traditional  test 
methods.  Use  of  ADS-enhanced  testing  on  open  air  ranges  may  be  limited  by  the  ability  to 
inject  the  synthetic  environment  into  the  platform. 

We  believe  the  intelligent  application  of  ADS  technology  for  testing  EW  systems  can  provide 
benefits  in  an  affordable  manner.  EW  program  and  test  managers  should  familiarize 
themselves  with  ADS  and  routinely  consider  its  use  in  their  deliberations  and  planning 
activities.  It  is  our  hope  that  ADS  will  be  treated  as  a  readily  available  tool.  While  the  use  of 
ADS  will  not  make  sense  in  every  case,  there  are  many  cases  where  it  not  only  makes  sense, 
but  it  may  offer  the  only  practical  approach  to  realistic  and  rigorous  testing  of  EW  integrated 
systems  and  systems  of  systems.  We  will  support  and  assist  program  and  system  managers  who 
see  applications  for  ADS  in  their  test  events  and  programs. 

Lust 

Philip  E.  T'ov/e  Richard’L.  Lockhart 

Director  Deputy  Director 

Operational  Test  and  Developmental  Test  and 

Evaluation  Evaluation 

OUSD  (A&TL) 


OUSD  (AT&L) 
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Executive  Summary 


1.0  Overview 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  by  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation),  Office  of  the  Secretary  of  Defense  (OSD)  (Acquisition  and  Technology)  in  October 
1994  to  investigate  the  utility  of  advanced  distributed  simulation  (ADS)1  technologies  for  support 
of  test  and  evaluation  (T&E).  The  JADS  program  is  Air  Force  led  with  Navy  and  Army 
participation  and  is  scheduled  to  end  in  March  2000.  This  report  addresses  the  third  of  three 
separate  JADS  tests,  the  Electronic  Warfare  (EW)  Test,  which  was  completed  in  April  1999. 

The  EW  Test  evaluated  the  utility  of  ADS  to  support  EW  T&E.  While  the  test  used  several 
efforts  to  examine  ADS-based  T&E,  the  cornerstone  effort  was  a  series  of  traditional  and  ADS- 
based  test  events  using  an  airborne  self-protection  jammer.  This  effort  was  called  the  self¬ 
protection  jammer  (SPJ)  test.  The  SPJ  test  defined  a  simple,  repeatable  test  scenario.  The 
scenario  was  executed  in  three  traditional  test  environments  to  create  a  data  baseline.  The  test 
scenario  was  then  executed  in  two  ADS-enhanced  test  environments.  The  first  ADS-based  test 
event  used  a  real-time  digital  system  model  (DSM)  interacting  with  manned  threat  simulators  at 
the  Air  Force  Electronic  Warfare  Environment  Simulator  (AFEWES)  facility.  The  second  ADS- 
based  test  used  the  SPJ  installed  on  an  F- 16  suspended  in  the  anechoic  chamber  at  the  Navy’s  Air 
Combat  Environment  Test  and  Evaluation  Facility  (ACETEF).  The  data  from  all  tests  were 
statistically  compared  in  an  attempt  to  quantify  the  impacts  of  ADS. 

The  other  efforts  used  by  JADS  to  examine  the  utility  of  ADS  to  support  EW  T&E: 

1)  The  OSD  CROSSBOW  Committee-sponsored  Threat  Simulator  Linking  Activity  (TSLA) 
effort. 

2)  The  Defense  Modeling  and  Simulation  Organization  (DMSO)-sponsored  High  Level 
Architecture  (HLA)  Engineering  Protofederation  (EPF)  effort,  and 

3)  The  Army’s  Advanced  Distributed  Electronic  Warfare  System  (ADEWS)  development  effort. 

Each  of  these  efforts  added  to  the  SPJ  test  experience  to  provide  JADS  with  a  broader 
understanding  of  the  utility  of  ADS  to  EW  T&E. 


1  Distributed  simulation-based  testing  can  be  accomplished  locally  as  is  practiced  today  at  hardware-in-the-loop 
(HITL)  facilities  and  installed  systems  test  facilities  (ISTFs)  such  as  the  Air  Force  Electronic  Warfare  Environment 
Simulator  (AFEWES)  and  the  Navy's  Air  Combat  Environment  Test  and  Evaluation  Facility  (ACETEF). 
Distributed  simulation-based  testing  can  also  be  geographically  distributed.  For  the  purpose  of  this  report,  ADS  is 
defined  as  a  networking  method  that  permits  the  linking  of  constructive  simulations  (digital  computer  models), 
virtual  simulations  (man-in-the-Ioop  or  hardware-in-the-loop  simulators),  and  live  players  located  at  distributed 
locations  into  a  single  test  environment/scenario. 


2.0  EW  Test  Results  and  Conclusions 

Within  the  confines  of  the  SPJ  test  data,  JADS  concluded  that  ADS  architectures  that  allow  the 
capabilities  of  geographically  separated  facilities  to  be  combined  to  create  a  realistic  test 
environment  for  EW  devices  can  be  designed.  This  allows  the  same  test  environment  to  be  used 
for  system  under  test  (SUT)  representations  ranging  from  DSMs  to  operational  equipment. 
Testing  in  a  common  ADS-based  environment  represents  a  significant  departure  from  the 
traditional  sequential  EW  test  process. 

Key  results 

•  Designing  ADS  architectures  requires  a  close  team  comprised  of  several  technical  experts 
spanning  several  disciplines  directed  by  a  system  integrator. 

•  The  architecture  produced  valid  results  for  both  the  DSM  and  actual  jammer  hardware. 

•  Latency  within  the  closed-loop  interaction  was  aggressively  managed,  and  JADS  was  able  to 
meet  its  objective  for  more  than  95%  of  the  runs. 

•  The  HLA  appears  to  be  a  feasible  method  for  linking  simulations  for  T&E.  It  is  appropriate 
to  use  HLA  to  link  to  other  HLA-compliant  simulations/simulators,  but  the  T&E  community 
should  not  view  it  as  the  only  solution  to  consider  in  designing  distributed  tests.  The 
selection  of  linking  technologies  should  be  driven  by  the  test  objectives.  In  many  cases, 
special  data  links  or  tactical  communications  links  may  be  more  appropriate  and  desired  for  a 
specific  test  objective. 

•  Two  of  the  eleven  EW  test  facilities  surveyed  in  1996  as  part  of  the  TSLA  effort  that  were 
appropriate  for  ADS-based  testing  have  been  closed.  While  this  is  a  significant  erosion  in  the 
infrastructure  needed  to  design  and  execute  ADS-based  tests,  it  already  limits  the  traditional 
EW  testing  process. 

3.0  Observations  for  EW  T&E 

JADS  assessment,  based  on  the  different  EW  Test  efforts,  is  that  ADS  has  varying  levels  of 
utility  for  EW  T&E.  These  levels  of  utility  depend  on  the  nature  of  the  EW  device  being  tested 
and  the  availability  of  suitable  test  facilities.  Single  function  EW  devices  and  federated  EW 
systems  are  expected  to  benefit  least  from  an  ADS-enhanced  test  process.  Only  radio  frequency 
jammers  may  see  sufficient  benefit  to  outweigh  the  additional  cost  of  an  ADS-enhanced  test 
process.  Integrated  EW  systems  may  see  significant  benefits  where  a  single  test  facility  is  not 
capable  of  providing  all  the  stimulation  (including  the  closed-loop  SUT  versus  manned  threat 
interaction  for  systems  that  include  radio  frequency  [RF]  jammers)  needed  to  simultaneously  test 
all  the  particular  integrated  EW  system’s  functions.  Systems  of  systems  testing  such  as  that 
required  for  electronic  support  (ES)  systems  should  see  significant  benefits  in  ADS-based  testing. 
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1.0  Purpose  And  Background 

1.1  Report  Purpose 

This  report  summarizes  the  assessment  of  the  utility  of  advanced  distributed  simulation2  (ADS) 
for  the  test  and  evaluation  (T&E)  of  electronic  warfare  (EW)  systems.  This  assessment  was 
based  on  the  results  and  lessons  learned  from  the  Joint  Advanced  Distributed  Simulation  (JADS) 
EW  Test  along  with  results  from  other  related  efforts. 

The  assessment  presented  in  this  report  provides  general  insight  into  the  implementation  of  ADS- 
based  testing  for  EW  systems.  More  detailed  guidelines  for  linking  specific  classes  of  systems 
for  EW  testing  can  be  found  in  the  Threat  Simulator  Linking  Activity  (TSLA)  specification  series 
and  in  the  JADS  special  reports. 

1.2  JADS  Overview 

The  JADS  Joint  Test  and  Evaluation  (JT&E)  program  is  an  Office  of  the  Secretary  of  Defense 
(OSD)-sponsored  joint  service  effort  designed  to  determine  how  well  an  emerging  technology, 
advanced  distributed  simulation  (ADS),  can  support  test  and  evaluation  activities.  The 
Department  of  Defense  (DoD)  has  always  used  rapidly  evolving  information  systems  technology 
to  support  its  needs.  Early  efforts  were  sharply  focused  on  training  applications  and  evolved 
from  the  Simulation  Network  (SIMNET)  program  managed  by  the  Advanced  Research  Projects 
Agency  (ARPA)  and  the  Army.  Conceptually,  the  early  projects  were  directed  toward  linking 
training  simulators  with  human  operators  at  distributed  geographical  sites  in  a  common  virtual 
environment.  The  players  interacted  with  one  another  in  this  common  environment  in  near  real 
time.  Over  the  years  SIMNET  has  evolved  into  a  technology  implementation  that  is  more 
flexible  and  robust  and  includes  different  types  of  simulators  with  different  levels  of  fidelity. 
(Reference  4)  The  capabilities  spawned  by  the  SIMNET  evolution  are  now  called  distributed 
interactive  simulation  (DIS)  and  are  documented  in  Institute  of  Electrical  and  Electronics 
Engineers  (IEEE)  Standard  1278.  The  high  level  architecture  (HLA)  is  the  latest  step  in  the 
effort  to  enable  DoD  simulations  to  connect  with  one  another  in  a  common  virtual  environment. 
In  1996.  Dr.  Paul  Kaminski,  Undersecretary  of  Defense  (Acquisition  and  Technology),  directed 
DoD  to  make  all  simulations  HLA  compliant,  although  it  is  not  yet  an  approved  IEEE  standard. 
(Appendix  B)  HLA  consists  of  an  interface  specification,  implementation  rules,  and  tools  to  help 
users  create  synthetic  environments  in  which  live,  virtual,  and  constructive  (synthetic)  players 
can  interact.  The  centerpiece  of  HLA  is  the  runtime  infrastructure  (RTI),  a  distributed  software 
application  that  handles  all  the  simulation-to-simulation  communication.  Because  of  widespread 


2  Distributed  simulation-based  testing  can  be  accomplished  locally  as  is  practiced  today  at  hardware-in-the-loop 
(HITL)  facilities  and  installed  systems  test  facilities  (ISTFs)  such  as  the  Air  Force  Electronic  Warfare  Environment 
Simulator  (AFEWES)  and  the  Navy’s  Air  Combat  Environment  Test  and  Evaluation  Facility  (ACETEF). 
Distributed  simulation-based  testing  can  also  be  geographically  distributed.  For  the  purpose  of  this  report,  ADS  is 
defined  as  a  networking  method  that  permits  the  linking  of  constructive  simulations  (digital  computer  models), 
virtual  simulations  (man-in-the-loop  or  hardware-in-the-loop  simulators),  and  live  players  located  at  distributed 
locations  into  a  single  test  environment/scenario. 
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interest  in  using  synthetic  environments  (and  the  technology  and  standards  needed  to  create 
them)  to  support  test  and  evaluation,  the  Air  Force  Operational  Test  and  Evaluation  Center 
(AFOTEC)  felt  that  a  JT&E  program  could  serve  as  an  exploratory  vehicle.  Accordingly,  the 
JADS  JT&E  program  was  nominated.  Interest  was  shared  by  both  the  developmental  and 
operational  test  communities.  The  services  concurred  in  the  need  for  rigorous  examination  of  the 
use  of  synthetic  environment  technology,  and  the  Deputy  Director,  Test,  Systems  Engineering 
and  Evaluation  (Test  and  Evaluation),  OSD  (Acquisition  and  Technology)  chartered  JADS  as  a 
joint  test  program  in  October  1994.  (Reference  1)  JADS  was  chartered  to  investigate  the  utility 
of  ADS  for  both  developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation 
(OT&E).  JADS  was  tasked  to  investigate  the  utility  of  ADS,  including  DIS  and  HLA,  for  T&E; 
to  identify  the  critical  concerns,  constraints,  and  methodologies  when  using  ADS  for  T&E;  and 
finally,  to  identify  the  requirements  that  must  be  introduced  in  ADS  systems  if  they  are  to  support 
a  more  complete  T&E  capability  in  the  future. 

JADS  investigated  ADS  applications  in  three  slices  of  the  T&E  spectrum:  the  System  Integration 
Test  (SIT)  explored  ADS  support  of  air-to-air  missile  testing;  the  End-to-End  (ETE)  Test 
investigated  ADS  support  for  command,  control,  communications,  computers,  intelligence, 
surveillance  and  reconnaissance  (C4ISR)  testing;  and  the  EW  Test  examined  ADS  support  for 
EW  testing.  The  JADS  Joint  Test  Force  (JTF)  was  also  chartered  to  observe  or  to  participate  at  a 
modest  level  in  ADS  activities  sponsored  and  conducted  by  other  agencies  in  an  effort  to  broaden 
conclusions  developed  in  the  three  dedicated  test  areas. 
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2.0  Supporting  Activities  and  Results 


The  EW  Test  was  built  on  four  discreet  efforts.  Each  effort  was  intended  to  provide  insight  into 
the  limitations  of  technology  supporting  ADS,  the  fundamental  requirements  that  ADS 
architectures  must  support  for  EW  T&E,  and  the  application  of  ADS  to  EW  testing.  These 
efforts  were 

1)  JADS-sponsored  and  managed  self-protection  jammer  test, 

2)  OSD  CROSSBOW  Committee-sponsored  Threat  Simulator  Linking  Activity, 

3)  Defense  Modeling  and  Simulation  Organization  (DMSO)-sponsored  High  Level  Architecture 
(HLA)  Engineering  Protofederation  (EPF),  and 

4)  Army-sponsored  Advanced  Distributed  Electronic  Warfare  System  (ADEWS)  development 
program. 

Each  effort  is  described  below  and  results  are  discussed  at  the  end  of  this  section. 

2.1  EW  Self-Protection  Jammer  (SPJ)  Test 

The  EW  SPJ  test  directly  evaluated  the  utility  of  ADS  to  support  testing  of  EW  systems  using  the 
ALQ-131  as  one  component  of  a  representative  EW  test  environment.  The  intent  was  to  recreate 
selected  test  events  in  the  development  cycle  of  an  airborne  self-protection  jammer  to  directly 
investigate  the  ability  of  ADS  to  address  perceived  shortfalls  in  the  EW  test  process  as  articulated 
in  A  Description  of  the  DoD  Test  and  Evaluation  Process  for  Electronic  Warfare  Systems, 
Revision  2,  dated  31  July  18,  1996,  prepared  by  the  Director,  Test,  Systems  Engineering  and 
Evaluation. 

The  SPJ  test  consisted  of  three  separate  test  phases.  The  first  test  phase  was  a  series  of 
traditional  tests  designed  to  collect  a  baseline  set  of  data.  During  this  test  series,  JADS  used  the 
Western  Test  Range  to  accomplish  14.4  hours  of  open  air  range  (OAR)  testing,  the  Air  Force 
Electronic  Warfare  Evaluation  Simulator  (AFEWES)  hardware-in-the-loop  (HITL)  facility,  and  a 
system  integration  laboratory  (SDL)  facility.  The  collected  data  were  used  to  calculate  ten 
traditional  EW  measures  of  performance  (MOPS).  These  MOPS  formed  the  baseline  data  set. 

The  second  and  third  SPJ  test  phases  recreated  the  baseline  test  environment  using  an  ADS 
architecture.  The  architecture  used  remained  constant  while  the  representation  of  the  SPJ 
changed.  The  intent  of  changing  the  SPJ  representation  was  to  mimic  testing  in  two  different 
SPJ  development  phases.  Phase  2  used  a  real-time  digital  system  model  (DSM).  Phase  3  used 
the  ALQ-131  mounted  on  the  host  aircraft  that  was  suspended  in  an  installed  systems  test  facility 
(ISTF).  In  both  Phase  2  and  Phase  3.  the  same  EW  data  were  collected  and  used  to  calculate  the 
same  ten  MOPs  calculated  in  Phase  1.  The  Phase  1  baseline  was  statistically  compared  to  the 
results  of  each  of  the  ADS-based  test  phases  to  quantify  impacts  on  the  EW  MOPs  caused  by 
using  ADS  to  create  a  virtual  test  environment.  Phase  2  and  Phase  3  are  discussed  below. 
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2.1.1  Phase  2  Digital  System  Model  Test 

JADS  wanted  to  demonstrate  that  ADS  could  be  used  to  create  a  common  test  environment  that 
could  potentially  be  used  throughout  system  under  test  (SUT)  development.  The  first 
representation  of  the  SUT  according  to  the  DoD  EW  test  process  is  the  DSM. 

The  test  process  discusses  using  DSMs  of  the  SUT  running  interactively  with  DSMs  of  “host 
platforms,  other  friendly  players,  the  combat  environment,  and  threat  systems  to  provide  cost 
effective  T&E.  (Reference  12,  Paragraph  2.4.2.2.7.1)  The  Phase  2  DSM  test  represented  a  test 
early  in  the  SPJ  development  that  is  an  extension  to  the  current  EW  test  process.  The  test 
process  extension  embodied  in  the  Phase  2  test  used  the  DSM  interacting  with  real  humans 
instead  of  models  to  gain  an  early  understanding  of  system  effectiveness  against  human 
operators.  In  the  JADS  test,  the  DSM,  hosted  at  the  Air  Combat  Environment  Test  and 
Evaluation  Facility  (ACETEF),  Patuxent  River  Naval  Air  Station,  Maryland,  interacted  in  a 
closed-loop  with  four  manned  threat  simulators  in  the  AFEWES  facility,  Fort  Worth,  Texas. 
Test  control  was  performed  at  the  JADS  facility  in  Albuquerque,  New  Mexico. 

The  DSM  test  used  a  distributed  test  architecture  to  recreate  the  baseline  OAR  and  HITL 
environments.  The  architecture  was  built  using  the  HLA  interface  and  RTI,  HLA  rules,  and  the 
available  HLA  tools.  The  architecture  linked  JADS  in  Albuquerque,  AFEWES  in  Fort  Worth, 
and  ACETEF  in  Patuxent  River.  AFEWES  provided  the  threats.  The  DSM  was  hosted  at 
ACETEF.  All  other  players  were  hosted  at  JADS.  Figure  1  shows  each  of  the  federates  and  the 
key  messages  each  sent  and  received. 
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TARGET  ASSIGNMENTS 


ENGAGEMENT  RESULTS 


AIRCRAFT  TSPI 


AIRCRAFT  TSPI 


ECM  =  electronic  countermeasures  IADS  =  Integrated  Air  Defense  System 

RF  =  radio  frequency  TSPI  =  time-space-position  information 

Figure  1.  EW  Test  Federates 

The  architecture  did  not  pass  radio  frequency  (RF)  waveforms  across  the  T-l  communication 
lines.  Instead,  operator  actions  were  translated  into  mode  change  messages  defined  in  the  JADS 
EW  Test  interface  control  document  (ICD).  The  AFEWES  threats  operate  in  RF;  however,  the 
DSM  is  obviously  not  capable  of  producing  RF.  Therefore,  JADS  used  equipment  at  AFEWES. 
the  JammEr  Technique  Simulator  (JETS),  to  generate  the  jamming  waveforms  based  on 
commands  from  the  DSM.  JADS  had  allocated  500  milliseconds  for  latency  in  the 
threat/jammer  closed-loop  interactions.  In  other  words,  JADS  wanted  no  more  than  250 
milliseconds  to  elapse  from  the  time  an  operator’s  actions  changed  the  state  of  the  threat  until  the 
mode  change  message  arrived  at  the  DSM  and  no  more  than  250  milliseconds  from  the  time  the 
DSM  issued  the  command  to  jam  until  RF  energy  came  from  the  JETS  at  AFEWES. 

The  DSM  test  completed  246  runs.  The  DSM  data  were  evaluated  by  subject  matter  experts  and 
deemed  valid  for  245  runs.  The  MOPs  did  not  show  a  consistent  degree  of  correlation  with 
either  the  OAR  or  HITL  data;  however,  this  was  not  due  to  ADS  effects.  Operator  response 
differences  at  both  the  OAR  and  HITL  and  threat  differences  between  AFEWES  and  the  OAR 
were  the  primary  variance  sources.  The  statistical  tests  JADS  used  also  contributed  to  the  lack  of 
correlation. 
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This  does  not  imply  that  ADS  had  no  effect  on  the  test.  Bursts  of  data  dropouts  in  the  time- 
space-position  information  (TSPI)  that  provided  the  aircraft/jammer  location  were  generally 
handled  by  dead  reckoning  algorithms  in  the  DSM  and  at  AFEWES.  The  algorithms  supplied  an 
estimated  aircraft  position  until  an  update  was  received.  At  the  point  TSPI  data  were  again 
received,  the  aircraft  was  moved  to  the  known  valid  position.  In  one  instance,  the  move  induced 
a  tracking  error  spike  greater  than  one  degree  in  azimuth  and  elevation  while  a  missile  was  in 
flight.  The  missile  was  not  able  to  recover,  resulting  in  an  invalid  miss  distance.  This  was 
observed  in  real  time  during  the  test  execution,*  and  the  data  were  scanned  for  similar 
occurrences. 

Other  potential  ADS  effects  such  as  latency  and  out-of-order  data  were  addressed  in  the  design  of 
the  architecture.  Only  eight  of  the  completed  runs  exhibited  latency  above  our  design  goal  for 
the  closed-loop  interaction.  Out-of-order  data  were  observed,  but  all  EW  MOP  calculations  were 
based  on  the  time  the  data  were  created,  not  when  they  were  received. 

2.1.2  Phase  3  Installed  Systems  Test  Facility  (ISTF)  Test 

The  Phase  3  ISTF  test  represented  another  extension  to  the  OSD  EW  test  process  that  would 
occur  very  late  in  the  SPJ  development.  In  this  case,  the  extension  was  closed-loop  testing  in  an 
ISTF  facility.  ISTF  test  events  use  “simulated  and  stimulated  inputs”  to  “provide  critical 
information  regarding  integrated  system  performance.”  (Reference  12,  Paragraph  2. 4.2.2. 5) 
However,  these  simulated  inputs  are  limited  to  those  that  the  ISTF  is  capable  of  generating.  In 
the  Phase  3  ISTF  test,  JADS  used  the  same  manned  threat  simulators  that  were  used  in  the  Phase 
2  DSM  test.  The  ISTF  at  ACETEF  did  not  have  manned  threat  simulators  for  these  threats. 
JADS  used  ADS  to  create  a  closed-loop  test  capability  that  allowed  JADS  to  collect  and  evaluate 
jammer  effectiveness  at  a  facility  that  had  no  manned  closed-loop  threat  simulators  for  the  threats 
of  interest. 

JADS  used  the  architecture  created  for  the  Phase  2  DSM  test  to  allow  the  real  jammer  to  interact 
with  the  AFEWES  threats.  The  actual  jammer  used  on  the  OAR  was  mounted  on  an  aircraft  and 
both  were  suspended  in  the  anechoic  facility  at  ACETEF.  The  aircraft  electronics  systems 
including  the  fire  control  radar  were  allowed  to  radiate  in  the  chamber  to  simulate 
electromagnetic  interference  (EMI)/compatibility  (EMC)  testing.  At  the  same  time,  the  threat 
signals  were  injected  into  the  jammer  and  the  jammer  responses  were  radiated  into  the  chamber. 
This  showed  that  it  may  be  possible  to  conduct  EMI/EMC  testing  in  conjunction  with  jammer 
effectiveness  testing,  providing  a  more  robust  test  environment. 

During  this  test  phase,  we  made  excursions  to  the  reference  test  condition  to  see  if  there  were  any 
readily  apparent  limitations  in  our  architecture.  During  these  runs,  ACETEF  used  the  Synthetic 
Warfare  Environment  Generator  (SWEG)  model  to  add  two  low  fidelity  threats  into  the  ACETEF 
environment.  This  was  a  simple  demonstration  of  how  threats  could  be  added  into  a  test 
environment  to  increase  the  threat  density. 

Again,  the  architecture  did  not  pass  RF  waveforms  across  the  T-l  communication  lines.  This 
time  the  pod  had  to  receive  RF  energy,  so  JADS  used  equipment  at  ACETEF,  the  Advanced 
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Tactical  Electronic  Warfare  Environment  Simulator  (ATEWES),  to  create  the  RF  energy  based 
on  AFEWES  operator  actions.  The  500  millisecond  latency  requirement  did  not  change. 
However,  more  equipment  at  ACETEF  was  involved  to  connect  the  actual  jammer  to  the  virtual 
environment  than  was  needed  to  connect  the  DSM  in  Phase  2. 

The  ISTF  test  completed  223  runs.  The  ISTF  data  were  evaluated  by  subject  matter  experts  and 
deemed  valid  for  all  223  runs.  The  MOPs  did  not  show  a  consistent  degree  of  correlation  with 
either  the  OAR  or  HITL  data.  Again  this  was  not  due  to  ADS  effects.  Operator  response 
differences  (at  both  the  OAR  and  HITL)  and  threat  differences  (between  AFEWES  and  the  OAR) 
were  the  primary  variance  sources.  The  statistical  tests  themselves  again  contributed  to  the  lack 
of  correlation.  However,  the  MOPs  showed  a  tendency  to  correlate  better  with  Phase  2  data  than 
with  any  other  data  set. 

Bursts  of  data  dropouts  in  the  TSPI  were  not  seen  and  latency  improved  overall.  A  new  effect 
was  seen  in  this  test.  We  had  to  abort  two  runs  when  the  time  synchronization  hardware  and 
software  failed  to  keep  AFEWES  synchronized. 

Other  potential  ADS  effects  such  as  latency  and  out-of-order  data  were  addressed  in  the  design  of 
the  architecture.  Only  one  of  the  completed  runs  exhibited  latency  above  our  design  goal  for  the 
closed-loop  interaction.  Out-of-order  data  were  observed,  but  all  EW  MOP  calculations  were 
based  on  the  time  the  data  were  created,  not  when  they  were  received. 

2.2  Threat  Simulator  Linking  Activity 

TSLA  was  an  OSD  CROSSBOW  Committee-sponsored  study  to  investigate  and  document  the 
requirements  and  potential  design  for  linking  existing  EW  test  resources.  This  study  was 
accomplished  in  three  phases.  The  first  phase  investigated  previous  and  ongoing  efforts  to  use 
ADS  in  EW  T&E  facilities.  Phase  2  reported  on  network  requirements  to  link  test  facilities. 
Phase  3  produced  preliminary  system/segment  specifications,  software  design  specifications  and 
interface  design  specifications  for  the  ADS  network.  This  effort  was  completed  in  1997  and  the 
results  can  be  obtained  from  CROSSBOW.  (References  4.  5.  6.  7.  8) 

2.3  HLA  Engineering  Protofederation  (EPF) 

In  1996,  DMSO  implemented  a  series  of  fast-paced  development  and  prototyping  efforts.  JADS 
monitored  the  EPF  that  evaluated  HLA  in  an  EW  T&E  environment.  This  effort  linked  facilities 
at  Patuxent  River,  Maryland;  Wright-Patterson  Air  Force  Base,  Ohio;  Buffalo,  New  York;  Fort 
Worth,  Texas;  and  Huntsville,  Alabama.  The  purpose  was  to  create  a  realistic  exercise  designed 
to  stress  the  HLA  federation  test  environment  to  realistic  T&E  levels.  The  results  are 
documented  in  evaluation  reports  produced  by  MITRE  and  by  ACETEF.  (References  13,  14) 
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2.4  Advanced  Distributed  Electronic  Warfare  System  (ADEWS) 

ADEWS  is  an  Army-sponsored  program  to  test  communications  systems  using  an  ADS-based 
jamming  system.  JADS  monitored  the  proof-of-principle  phase  in  which  the  Army  completed  a 
field  demonstration  of  their  virtual  jamming  system.  This  system  is  comprised  of  a  Covert 
Remote  Electronic  Warfare  Simulator  (CREWS)  transmitter  mounted  on  the  victim 
communications  equipment  and  connected  between  the  antenna  and  the  equipment.  The 
CREWS  transmitter  is  controlled  by  a  jammer  simulator  that  uses  RF  tones  to  communicate 
wave-form  and  transmitted  power  to  each  device.  Received  power  is  determined  within  the 
CREWS  device  by  calculating  the  distance  and  terrain  effects  between  the  jammer  and  the 
victim. 

2.5  Summary  of  Results 
2.5.1  SPJ  Test  Results 

The  key  results  from  the  SPJ  testing  were  as  follows. 

-  Designing  ADS  architectures  requires  a  close  team  comprised  of  several  technical  experts 
spanning  several  disciplines  directed  by  a  system  integrator.  Expertise  required  includes 
SUT,  test  facility,  instrumentation,  data  collection  and  analysis,  wide  area  computer 
communications,  local  area  computer  communications,  computer  operating  system  and 
communications  protocol,  and  RTI  (or  other  linking  technology). 

—  The  architecture  produced  valid3  results  for  both  the  DSM  and  actual  jammer  hardware.  Few 
ADS-induced  errors  were  observed.  These  were  easily  identified  and  the  data  were  removed 
from  the  valid  data  set. 

—  Latency  within  the  closed-loop  interaction  was  aggressively  managed,  and  JADS  was  able  to 
meet  its  objective  for  more  than  95%  of  the  runs.  Variance  in  latency  may  be  more 
problematic  for  some  test  designs.  JADS  time  stamped  each  message  before  transmission 
and  used  this  time  stamp  for  analysis  instead  of  the  time  of  receipt. 

-  Data  loss  and  out-of-order  data  are  ADS  effects  that  must  be  addressed  in  the  architecture 
design.  Assigning  unique  message  numbers  and  the  time  stamps  applied  before  transmission 
to  detect  and  deal  with  data  loss  and  out-of-order  data  was  effective. 

—  Bursts  of  data  loss  were  seen  in  the  TSPI  data  stream  in  the  Phase  2  DSM  architecture; 
however,  the  losses  generally  did  not  affect  the  runs  because  both  the  DSM  and  AFEWES 
used  dead  reckoning  algorithms.  This  allowed  both  players  to  estimate  the  SUT  position 
until  the  TSPI  resumed. 

-  These  losses  were  still  less  than  1%  of  all  the  messages  transmitted.  The  Phase  3  ISTF 
architecture  used  a  later  RTI  release  and  minor  changes  were  made  in  the  network 
hardware.  The  bursts  of  data  loss  were  not  seen  in  the  TSPI  data  stream  in  Phase  3. 


1  Data  validity  was  determined  by  subject  matter  experts.  The  expert  panel  determined  that  the  data  were  consistent 
with  the  expected  results  from  traditional  tests  using  the  same  reference  test  condition.  Direct  statistical  comparison 
was  inconclusive  primarily  because  of  differences  in  operator  skill  levels. 
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-  The  ADS  architecture  provided  ample  bandwidth. 

-  Network  component  failures  interrupted  testing  three  times  for  less  than  15  minutes  total  test 
time.  Computer  crashes  at  JADS  and  hardware  failures  within  ACETEF  and  AFEWES  were 
responsible  for  most  of  the  aborted  runs  and  lost  test  time,  respectively. 

2.5.2  Threat  Simulator  Linking  Activity  Results 

-  The  SPJ  test  validated  key  aspects  of  the  TSLA  specification  set.  Validation  included 
message  sizes,  data  packing  in  message  structures  for  increased  efficiency,  and 
instrumentation  required  to  accomplish  ADS  testing. 

-  The  TSLA  resources  survey  accomplished  in  1996  examined  different  aspects  of  eleven  EW 
test  facilities  to  support  ADS  for  testing  EW  systems.  Two  of  these  facilities  no  longer  do 
EW  testing;  one  was  closed  and  relocated,  and  one  is  planned  to  be  closed  and  relocated.  The 
Air  Force  has  been  impacted  the  most  by  these  changes.  The  Air  Force-owned  facilities  that 
were  not  affected  by  the  changes  are  the  least  experienced  in  ADS. 

2.5.3  HLA  Engineering  Protofederation  Results 

-  JADS  used  the  EPF  ICD  as  starting  point  for  developing  the  JADS  EW  Test  ICD. 

-  The  HLA  appears  to  be  a  feasible  method  for  linking  simulations  for  T&E.  Direct  results 
from  the  EPF  effort  provided  JADS  with  a  baseline  for  measuring  how  quickly  HLA  is 
maturing.  It  is  currently  immature:  however,  it  is  evolving  rapidly.  The  pace  of  change 
coupled  with  its  immaturity  made  it  difficult  to  use. 

-  The  runtime  infrastructure  (RTI)  is  also  immature  although  it  has  significantly  improved. 
The  SPJ  test  results  indicate  that  the  current  DMSO-provided  RTI  effectively  moves  data,  but 
the  lack  of  documentation  and  the  latency  variation  added  by  the  RTI  are  issues.  The  runtime 
infrastructure  improved  throughout  the  JADS  SPJ  test.  JADS  used  a  later  version  of  the  RTI 
in  the  Phase  3  ISTF  test  and  saw  improvements  in  latency  performance  and  a  significant 
reduction  in  lost  data. 

•  It  is  appropriate  to  use  HLA  to  link  to  other  HLA-compliant  simulations/simulators,  but  the 
T&E  community  should  not  view  it  as  the  only  solution  to  consider  in  designing  distributed 
tests.  Facilities  such  as  AFEWES  and  ACETEF  have  built  HLA-compliant  interfaces,  but 
communications  between  resources  within  their  facilities  are  done  using  whatever  approach 
makes  the  most  sense.  The  selection  of  linking  technologies  should  be  driven  by  the  test 
objectives.  In  many  cases,  special  data  links  or  tactical  communications  links  may  be  more 
appropriate  and  desired  for  a  specific  test  objective. 

2.5.4  Advanced  Distributed  Electronic  Warfare  System  Results 

-  The  proof  of  concept  was  successful  and  further  developments  are  being  pursued.  Expansion 
of  the  current  CREWS  device  to  test  global  positioning  system  (GPS)  jamming  is  expected  to 
be  funded  in  fiscal  year  (FY)  00. 
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3.0  Overall  ADS  Utility  Assessment 
3.1  JADS  Issues 


The  JADS  JT&E  program  was  chartered  to  investigate  the  utility  of  ADS  for  both  DT&E  and 
OT&E.  The  charter  letter  identifies  three  issues  to  be  addressed. 

-  Investigate  the  present  utility  of  ADS,  including  DIS  for  T&E.  The  utility  assessment 
includes  evaluating  the  validity  of  data  from  tests  using  ADS  and  the  benefits  of  using  ADS 
in  T&E. 

-  Identify  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E. 

-  Identify  the  requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support  a 
more  complete  T&E  in  the  future. 

The  ability  of  ADS  to  support  EW  T&E  will  be  assessed  in  terms  of  these  issues. 

3.2  General  Utility  of  ADS  for  Electronic  Warfare  T&E 

3.2.1  General  Utility  Assessment 

ADS  has  utility  for  EW  DT&E  and  OT&E  because  ADS-supported  tests  can  provide  valid  EW 
system  performance  evaluations  in  a  number  of  areas  addressed  by  both  DT&E  and  OT&E  (see 
section  3.2.3),  and  because  there  are  benefits  to  using  ADS  for  EW  T&E  (see  section  3.2.2).  The 
application  of  ADS  for  a  specific  EW  system  (see  section  4.2)  depends  on  (1)  the  test  objectives, 
(2)  the  characteristics  of  the  EW  system,  (3)  the  availability  of  EW  DSMs,  sensor  stimulators, 
HITL  or  OAR  manned  threat  simulators,  and  (4)  the  details  of  the  test  scenario.  Simple 
guidelines  for  designing  an  architecture  for  ADS-supported  testing  are  contained  in  Appendix  C. 

3.2.2  General  ADS  Benefits 

ADS  allows  the  test  designer  to  break  the  traditional  mold  of  stovepipe  sequential  testing.  The 
following  are  the  general  benefits  ADS  brings  to  EW  system  development  and  EW  T&E. 

-  ADS  supports  more  robust  validation  of  DSMs  by  allowing  DSMs  to  interact  with  manned 
simulators. 

-  This  provides  more  confidence  in  model-on-model  results. 

-  ADS  extends  the  use  of  secure  test  environments  beyond  the  boundary  of  a  single  facility. 

-  ADS  allows  the  use  of  common  threats  and  environments  across  development  phases. 

-  Common  threats  and  environments  promote  the  one-to-one  comparison  of  results  across 
development  phases. 

-  Common  threats  and  environments  couple  well  with  the  Navy’s  concept  of  scenario- 
driven  T&E. 

-  ADS  allows  reactive  scenarios  to  be  used  across  all  development  phases. 
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-  ADS  allows  operationally  relevant  testing  to  occur  earlier  in  development. 

-  ADS  facilitates  testing  integrated  EW  systems  and  systems  of  systems. 

—  ADS  allows  direct  measurement  of  mission-level  MOPs  and  measures  of  effectiveness 
(MOEs). 

-  ADS  allows  tests  to  address  broader  testing  objectives  (e.g.,  mission  level)  beyond  the 
capability  of  a  single  facility. 

3.2.3  General  Role  of  ADS  in  EW  T&E 

ADS  allows  the  test  designer  to  construct  tests  that  address  objectives  beyond  the  capability  of  a 
single  facility.  While  connecting  facilities  can  be  beneficial,  not  all  EW  testing  requires  the  use 
of  ADS.  Single  device  and  federated  systems  testing  are  generally  adequately  supported  by  the 
sequential  facility  testing  paradigm  (the  progression  from  DSM  to  SIL,  SIL  to  HITL,  HITL  to 
ISTF,  ISTF  to  OAR)  recommended  in  the  DoD  EW  test  process  description.  Testing  integrated 
EW  systems  and  systems  of  systems  does  not  readily  conform  to  the  sequential  facility  testing 
paradigm.  The  requirement  for  testing  these  systems  is  to  create  an  operationally  realistic  single 
environment  in  which  all  integrated  EW  functions,  including  closed-loop  interaction  with 
manned  threat  simulators,  are  simultaneously  tested  while  interacting  with  the  host  platform,  the 
host  platform  operator(s),  and  the  appropriate  blue  and  red  players.  Any  less  stringent  test 
increases  the  risk  of  fielding  a  flawed  system.  Highly  integrated  avionics  systems  such  as  the  F- 
22  share  resources.  The  only  way  to  fully  understand  the  effects  of  resource  sharing  is  to  test  all 
system  functions  at  the  same  time  in  the  operationally  realistic  single  environment  described 
above. 

To  be  tested,  all  integrated  EW  systems  may  not  require  ADS.  If  a  single  facility  is  capable  of 
providing  all  the  stimulation  (including  the  closed-loop  SUT  versus  manned  threat  interaction  for 
systems  that  include  RF  jammers)  for  the  particular  integrated  E3V  system,  then  ADS  is  not 
needed.  However,  if  no  facility  exists,  ADS  offers  an  alternative  to  potentially  expensive  facility 
modifications  or  expansion.  The  development  and  integration  of  distributed  test  facilities  (such 
as  AFEWES  and  ACETEF)  capable  of  simultaneous  EW  sensors  stimulation  and  closed-loop 
threat  interaction  presents  the  facility  designer  with  the  same  problems  of  latency,  data  loss,  and 
out-of-order  data  that  confront  the  ADS  architecture  designer. 

Systems  of  systems  testing  is  not  likely  to  be  practical  or  affordable  in  a  single  facility  or  on  a 
single  OAR.  Simple  cooperative  jamming  tests  against  single  threats  can  be  accomplished  using 
the  sequential  testing  methods  described  in  the  OSD  EW  test  process.  More  complicated 
systems  of  systems  testing  on  an  OAR  is  constrained  by  the  limitations  the  single  facility  or  the 
limitations  of  the  range  selected  and  the  cost  of  bringing  the  required  platforms  together  for  the 
test  event.  The  JADS  End-to-End  Test  was  a  command,  control,  communications,  computers 
and  intelligence  (C4I)  systems  of  systems  test  using  the  Joint  Surveillance  Target  Attack  Radar 
System  (Joint  STARS).  JADS  concluded  “ADS  has  great  potential  as  a  C4ISR  testing  tool  and 
provides  a  viable  means  of  conducting  realistic  mission-level  evaluations.”  (Reference  3) 

Some  OAR  limitations  can  be  removed  and  OAR  environments  can  be  extended  through  the  use 
of  ADS.  The  ADEWS  program  demonstrated  that  ADS  allows  realistic  large-scale 
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communications  jamming  without  impacting  commercial  communications.  The  JADS  EW  Test 
investigated  and  concluded  that  ADS  can  be  used  to  link  manned  simulators  of  semi-active 
threats  on  the  OAR  to  HITL-based  seeker  simulators.  This  arrangement  allows  the  seeker 
simulator  to  use  the  OAR  track  loop  errors  while  it  operates  in  a  realistic  (correct  Doppler)  RF 
environment.  (The  architecture  needed  to  accomplish  this  is  similar  to  the  SIT  live  fly 
architecture.)  By  using  real-time  links,  the  OAR  can  provide  real-time,  high  fidelity,  semi-active 
missile  flyouts  complete  with  OAR  tracking  loop  error  effects.  (Reference  9) 

3.3  Critical  Constraints,  Concerns,  and  Methodologies  for  EW  T&E 

3.3.1  Geographically  Distributed  Interactive  Simulation  Infrastructure  Limitations 

Certain  constraints  limit  the  applicability  of  ADS  to  EW  T&E.  Some  of  these  constraints  are  due 
to  the  DSMs,  HITL  facilities,  ISTF  facilities,  or  OARs  and  are  common  constraints  for  both 
ADS-supported  testing  and  testing  using  the  OSD  EW  test  process.  Other  constraints  result  from 
linking  requirements  and  are  related  to  ADS  implementation.  The  constraints  are  summarized  as 
follows. 

•  DSM  Constraints 

-  DSMs  must  work  in  real  time  to  usefully  link  with  manned  threat  simulators,  manned 
operator  stations,  or  other  real-time  simulations  of  blue  or  red  players.  Real-time 
execution  is  not  a  requirement  for  HLA  or  Joint  Modeling  and  Simulation  System 
(JMASS)  compliance.  Unlike  HLA,  JMASS  is  not  currently  capable  of  real-time 
execution. 

•  HITL,  ISTF  and  OAR  Constraints 

-  While  ADS  provides  the  ability  to  use  resources  across  facilities,  the  fidelity  of  the 
resulting  environment  is  limited  by  the  fidelity  of  the  test  infrastructure  to  create  each 
piece  of  the  environment.  Specifically,  the  fidelity  of  the  synthetically  generated  signals 
is  limited  to  the  capability  of  the  simulator/stimulator  connected  to  the  SUT,  threat,  or 
other  players  in  the  scenario.  (This  is  not  unique  to  ADS  test  environments.  The  fidelity 
of  the  environment  is  always  a  test  constraint  and  must  be  addressed  in  the  test  design 
analysis  as  well  as  the  post-test  data  analysis  and  evaluation  of  results.) 

-  Understanding  resource  and  facility  limitations  is  critical  to  test  design.  Each 
resource/facility  brings  its  inherent  errors,  limitations,  and  assumptions  into  the  test 
architecture.  These  errors,  limitations,  and  assumptions  must  be  identified  and  managed 
in  the  test  design  to  avoid  problems  with  increased  data  variance  and/or  decreased  test 
validity.  Verification  and  validation  (V&V)  of  the  players  and  the  architecture  are 
essential.  (Again,  this  is  not  unique  to  ADS  test  environment  designs.  However,  it  is 
more  important  that  these  limitations  are  known  up  front  to  ensure  that  the  best  quality 
environment  is  created.  Connecting  facilities  tends  to  highlight  their  limitations.) 

-  The  TSLA  specifications  discuss  the  requirement  for  additional  types  of  quality  assurance 
instrumentation  to  monitor  the  environment  representation  at  each  location.  Subtle 
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differences  in  the  waveform  representations  among  locations  have  the  potential  to  change 
how  each  player  behaves  in  the  scenario  and  may  impact  the  test  measures. 

-  The  reliability  of  each  test  architecture  component,  resource,  or  facility  has  to  be  factored 
into  the  test  design.  The  expected  reliability  of  an  ADS  architecture  is  the  product  of  the 
expected  reliability  of  each  component,  resource,  and  facility.  As  such,  reliability  is 
expected  to  decrease  as  the  number  of  components  or  players  increase. 

“  OAR  threats  to  be  included  in  a  synthetic  environment  require  target,  electronic 
countermeasures  (ECM),  and  clutter  signal  injection  capabilities.  These  capabilities  are 
necessary  for  the  threats  to  react  correctly  to  synthetic  players.  OAR  threats  are  necessary 
because  of  the  limited  number  of  manned  threat  simulators  available  in  HITLs  and  ISTFs. 
Executing  EW  testing  using  the  OSD  EW  test  process  is  already  limited  by  the 
availability  of  suitable  simulators  and  environment  representations  in  HITLs  and  ISTFs. 


•  ADS-Related  Constraints 

-  Latency  is  a  limitation  on  how  tightly  two  players  can  be  coupled.  The  Phase  2  DSM  and 
Phase  3  ISTF  architectures  were  capable  of  average  round-trip  transmission  latency  for 
HLA  “reliable”  (transmission  control  protocol  [TCP]/Intemet  protocol  [IP])  interactions 
of  254  and  167  milliseconds  respectively.  “Best  effort”  (user  datagram  protocol  [UDP] 
multicast)  latency  was  considerably  better,  and  the  architecture  could  have  supported 
round-trip  transmission  latency  of  less  than  100  milliseconds  if  JADS  had  used  this 
protocol. 

-  Commercial  telecommunications  technology  and  computer  communications  protocols 
used  by  JADS  (as  well  as  DIS  and  HLA)  do  not  support  the  transmission  of  native 
spectrum  environment  data.  Techniques  currently  used  for  transmitting  analog 
electromagnetic  waves  in  local  distributed  simulation  facilities  such  as  RF  waveguide 
and  fiber  optic  links  are  not  affordable  for  geographically  separated  facilities.  While 
analog  waveforms  can  be  captured  digitally  and  transmitted  over  commercial 
telecommunications  lines  using  computer  communications  protocols,  they  can  use 
considerable  bandwidth.  JADS  did  not  find  bandwidth  to  be  an  issue.  However,  if  entire 
digitized  waveforms  are  being  transmitted,  bandwidth  will  quickly  become  an  issue. 

-  Data  loss  must  be  addressed  in  test  design.  JADS  lost  far  less  than  1%  of  data 
transmitted,  however  data  loss  depends  on  several  factors.  JADS  EW  Test  showed  that 
the  lowest  latency  computer  communications  protocol  consistently  showed  the  highest 
data  loss.  Data  loss  impacts  can  be  reduced  by  using  error  correction  such  as  dead 
reckoning  to  replace  data  missing  from  a  TSPI  stream.  Event  data  can  not  be  predicted. 
Instead,  event  data  must  be  either  sent  using  more  reliable  but  higher  latency  computer 


JThe  JADS  experience  indicates  that  the  Air  Force  test  process  described  in  Air  Force  Manual  (AFMAN)  99-112, 
Electronic  Warfare  Test  and  Evaluation  Process  Direction  and  Methodology  for  EW  Testing,  is  likely  not 
executable  with  Air  Force  EW  test  resources.  Shortfalls  include  HITL  threat  representations  for  all  likely  threats  of 
interest  and  limited  ISTF  threat  representations.  Including  the  use  of  Navy  facilities  may  ease  the  ISTF  limitations. 
This  is  a  significant  issue  for  Air  Force  EW  testing  and  for  implementation  of  ADS-based  testing.  Closing  and 
relocating  the  sole  Air  Force  high  fidelity  threat  simulator  HITL  as  recommended  in  the  Air  Force’s  1996  Electronic 
Combat  Master  Plan  and  reiterated  in  the  1999  DRMD  912  Congressional  Report  is  likely  to  reduce  the  Air  Force’s 
ability  to  execute  AFMAN  99-12. 
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communications  protocols,  transmitted  multiple  times,  transmitted  with  periodic  data 
from  the  same  player,  or  event  data  losses  must  simply  be  accepted. 

-  Out-of-order  data  are  another  ADS  effect  that  must  be  addressed  in  test  design. 
Differences  in  transmission  methods,  differences  in  distance,  and  the  one-device-at-a- 
time  nature  of  some  computer  communications  protocols  are  all  contributing  factors. 
Designers  need  to  be  aware  of  the  sources  and  deal  with  them  in  the  design  of  the  test 
architecture,  data  collection,  and  data  analysis. 

There  are  concerns  which  must  be  addressed  for  the  proper  implementation  of  ADS. 

-  The  cost  of  implementing  ADS.  JADS  found  the  single  largest  cost  for  implementing 
ADS  to  be  the  cost  of  modifications  to  facilities  to  allow  them  to  connect.  This  cost  may 
be  significant  (e.g.,  about  $2  million  for  the  EW  SPJ  test).  Early  testing  against  HITL 
threats  is  likely  to  increase  cost  over  the  current  OSD  EW  test  process  design.  However, 
early  testing  against  HITL  threats  should  uncover  problems  earlier  in  the  development 
cycle.  Ultimately,  increased  costs  will  have  to  be  weighed  against  the  improved  test 
realism  benefit,  the  potential  improvements  in  test  capability  through  combining 
facilities,  and  the  potential  of  cost  avoidance. 

-  Use  of  dedicated  links.  The  use  of  dedicated  leased  telecommunications  lines  may  be 
justified,  rather  than  using  existing  networks  such  as  Defense  Simulation  Internet  (DSI), 
because  of  latency,  reliability,  and  scheduling  requirements.  Each  linking  application 
must  evaluate  its  requirements  and  justify  the  use  of  commercial  links  when  appropriate. 
(Reference  2) 

There  are  cultural  and  business  practice  concerns  that  must  be  addressed  for  the  proper 
implementation  of  ADS. 

-  Segments  of  the  EW  community  do  not  accept  ADS  as  a  “valid”  way  to  test  because  it 
does  not  conform  to  the  traditional  approach.  The  JADS  SPJ  test  lacked  the  complexity 
to  convince  many  in  this  group.  Others  have  not  heard  about  the  SPJ  test  despite  JADS 
attempts  to  reach  the  community.  As  a  result,  select  EW  test  facilities  are  driving  the 
demand  and  advances  in  ADS-based  testing  and  test  design.  Some  of  these  facilities  have 
been  more  successful  in  their  attempts  at  implementing  ADS  than  others.  However,  the 
EW  system  designers  and  testers  are  largely  silent  and  are  not  placing  requirements  for 
ADS-based  testing  on  the  test  facilities.  This  last  observation  is  based  on  JADS 
interactions  with  selected  Air  Force  and  Navy  test  facilities  and  interactions  with  EW 
development  and  test  organizations. 

-  Facilities  are  challenged  when  implementing  ADS-based  tests.  Most  existing  facilities 
were  designed  to  perform  stand-alone  tests.  Facility  details  such  as  the  ability  to  time 
synchronize  the  entire  facility  and  internal  latencies  that  are  not  a  factor  in  traditional  tests 
are  critical  in  ADS-based  tests.  This  forces  facilities  to  investigate  design  details  that 
may  have  been  made  decades  earlier,  and  it  forces  facilities  to  reveal  more  of  their 
internal  workings  than  they  are  accustomed  to  revealing. 

-  ADS  testing  requires  skills  not  found  in  traditional  testing.  Nontraditional  skills  include 
wide  area  computer  network  design,  integration,  test,  and  operation;  local  area  computer 
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network/wide  area  computer  network  integration  and  optimization;  computer/simulation 
interface  design,  integration,  optimization,  and  test;  and  if  HLA  is  used, 
RTI/computer/local  area  network/wide  area  network  installation,  optimization,  and 
operation. 

A  number  of  methodologies  apply  for  ADS  implementation.  These  are  discussed  in  JADS 
special  reports  and  are  simply  listed  here. 

-  Test  planning  (including  cost  benefit  analysis) 

-  ADS  architecture  and  instrumentation  design 

-  Verification  and  validation  (V&V)  of  the  ADS  configuration 

-  Integration  of  linked  assets 

-  Test  control 
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4.0  ADS  Applicability  to  T&E  of  Specific  EW  Types 
4.1  ADS  Applicability  Assessment  Approach 

Generalizing  the  results  of  the  JADS  EW  Test  to  all  possible  EW  systems  operating  in  all 
possible  environments  is  difficult.  The  JADS  experience  with  an  airborne  self-protection 
jammer  is  limited  when  compared  with  all  the  possible  types  and  operational  spectra  of  EW 
systems.  JADS  gained  direct  experience  with  only  four  EW  test  facilities.  While  the  TSLA 
survey  provided  a  larger  experience  base,  it  was  limited  to  RF  and  several  political  decisions 
have  significantly  changed  the  test  infrastructure  base  to  support  EW  testing.  Additionally, 
several  facilities  are  planning  or  executing  upgrades.  Therefore,  the  following  is  an  extrapolation 
based  on  the  JADS  results. 

JADS  elected  to  assess  applicability  of  ADS  testing  to  specific  EW  functions  by  examining  the 
nature  of  the  test  objective  with  respect  to  the  interaction  of  the  EW  function  with  the  threat.  Air 
Force  Manual  99-112,  Electronic  Warfare  Test  and  Evaluation  Process  -  Direction  and 
Methodology  for  EW  Testing,  identifies  two  types  of  interactions  between  the  EW 
function/device  with  the  threat.  The  first  interaction  is  “closed  loop.”  Closed-loop  testing  is 
defined  as  “A  form  of  EW  testing  in  which  both  the  friendly  and  threat  systems  react  to  each 
other’s  actions.”  While  many  “loops”  can  be  closed  in  an  EW  test  within  the  EW  system  or  host 
platform,  this  definition  highlights  the  interaction  of  the  EW  system  with  the  threat.  RF  jammers 
are  the  best  example  of  an  EW  function  that  is  tested  in  a  closed  loop.  The  second  interaction  is 
“open  loop.”  Open-loop  testing  is  defined  as  “A  test  scenario  in  which  a  system  reacts  to 
another’s  actions  without  resulting  feedback.”  EW  receivers  are  the  best  example  of  an  EW 
function  that  is  tested  open  loop.  RF  jammers  can  be  tested  open  loop  to  verify  specification 
compliance.  Jammer  response  time,  percent  correct  response,  and  frequency  accuracy  are  all 
examples  of  measures  that  can  be  collected  in  open-loop  testing.  However,  jammer  effectiveness 
measures  such  as  tracking  error  and  miss  distance  require  closed-loop  tests.  Focusing  on  the 
nature  of  the  device  interaction  highlights  the  benefits  and  limitations  of  ADS  testing  for  a  class 
of  devices  without  addressing  technical  details  of  the  test  or  SUT. 

The  distinction  between  open-loop  and  closed-loop  testing  in  an  ADS  environment  has 
significant  impact  on  the  architecture  and  test  design.  Closed-loop  tests  are  more  susceptible  to 
latency  which,  in  turn,  limits  the  actions  that  can  be  taken  to  address  data  loss  and  out-of-order 
data.  However,  closed-loop  tests  are  essential  to  understand  the  effectiveness  of  RF  jammers. 

While  the  open-loop/closed-loop  distinction  between  EW  tests  is  important,  further  distinctions 
are  needed  to  understand  the  advantages  and  limitations  of  ADS-based  testing  for  EW.  JADS 
selected  three  representations  of  the  SUT  to  examine  where  ADS  may  be  advantageously 
applied.  These  representations  are  ( 1 )  DSM,  (2)  uninstalled  hardware  (breadboard,  brassboard, 
final  hardware  traditionally  tested  in  SILs  and  HITLs),  and  (3)  installed  hardware  (hardware 
integrated  in  either  a  test  bed  platform  or  an  instrumented  version  of  the  target  platform 
traditionally  tested  in  ISTFs  or  on  OARs).  These  representations  cover  the  entire  range  of  SUT 
maturity  during  development. 
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This  approach  was  selected  to  emphasize  the  application  of  ADS  to  testing  EW  systems  without 
focusing  on  the  specific  function  or  functions  of  the  device,  the  spectrum  of  the  device,  or  the 
platform  hosting  the  device.  Specific  EW  functions  were  examined  to  ensure  that  the  approach 
accurately  addressed  the  benefits  of  using  ADS-based  testing.  EW  functions  examined  included 
warning  receivers  (RE  and  directed  energy),  intelligence  receivers  (RF  and  directed  energy), 
jammers  (self-protection  RF  and  non-RF,  standoff  RF  and  non-RF,  and  communications), 
countermeasure  dispensers,  offboard  countermeasures  (chaff,  flares,  smoke,  other  decoys). 
JADS  noted  that  single  function  devices  were  different  than  systems  that  integrated  multiple 
functions.  These  are  addressed  separately  in  Table  1.  Similarly  platforms  had  to  be  investigated 
to  ensure  that  the  results  remained  valid.  Platforms  examined  include  aircraft,  ground  vehicles, 
surface  vessels,  submarines,  and  satellites.  While  not  all  facilities  were  used  for  all  types  of 
platforms  (there  are  no  indoor  installed  system  test  facilities  for  Navy  ships,  for  example)  the 
discussion  is  sufficiently  broad  to  allow  the  reader  to  apply  these  results. 

4.2  ADS  Applicability  Assessment  Results 

Table  1  summarizes  JADS  assessment  of  ADS  benefits  and  the  utility  of  ADS  for  open-loop 
single  function  devices;  open-loop  integrated  EW  systems;  closed-loop,  single-function  devices; 
and  closed-loop  integrated  EW  systems  in  all  development  configurations  from  DSM  through 
breadboard  and  brassboard  to  installed  equipment.  All  benefits  assume  that  the  designer  has 
accomplished  an  analysis  of  player  interactions,  has  identified  those  interactions  that  least 
tolerate  latency  and  has  determined  that  an  architecture  can  be  constructed  that  can  meet  the 
latency  requirement. 
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Table  1.  ADS  Applicability  Assessment  Results 


Test 

Objective 

Type 

EW 

System 

Type 

Current  DoD 
Test  Process 
Facility 

ADS  Benefits 

Utility  Assessment 

Open  Loop 

Single 

Device 

DSM 

None 

Limited. 

-  Model-on-model  interactions 
should  be  capable  of  generating 
the  correct  environment.  (No 
investigation  of  model 
suitability  was  made.) 

-  Real-time  DSMs  may  allow 
dissimilar  alternatives  to  be 
directly  compared  during 
analysis  of  alternatives  phase  of 
development. 

Benefits  are  not  expected  to 
overcome  cost  of  ADS  for  this 
application. 

Single 

Device 

Uninstalled 

SIL,  HITL 

Limited. 

-  Local  stimulators  should  have 
the  ability  to  generate  the 
correct  environment  or  the  SUT 
is  untestable  at  that  location. 

-  ADS  adds  the  ability  to 
remotely  control  some  players. 

Benefits  are  not  expected  to 
overcome  cost  of  ADS  for  this 
application. 

Single 

Device 

Installed 

! 

ISTF,  OAR 

Limited. 

-  Local  stimulators  should  have 
the  ability  to  generate  the 
correct  environment  or  the  SUT 
is  untestable  at  that  location. 

-  ADS  adds  the  ability  to 
remotely  control  some  players. 

Benefits  are  not  expected  to 
overcome  cost  of  ADS  for  this 
application. 

Integrated 
EW  System 
DSM 

None 

Moderate. 

-  Model-on-model  interactions 
should  be  capable  of  generating 
the  correct  environment.  (No 
investigation  of  model 
suitability  was  made.) 

-  Real-time  DSMs  facilitate 
operator  interactions  in  model- 
on-model  efforts.  Useful  if 
models  can  be  reactive. 

-  Real-time  DSMs  may  allow 
dissimilar  alternatives  to  be 
directly  compared  during 
analysis  of  alternatives  phase  of 
development. 

Benefits  may  offset  the  cost  of 
ADS  architecture  and  real-time 
DSM  development. 
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Test 

Objective 
Ty 


Integrated 
EW  System 
Uninstalled 


Current  DoD 
Test  Process 
Facility 


SIL,  HITL 
(Testing  must 
place  all 
sensors  in  the 
same 

environment 
at  the  same 
time.) 


Integrated 
EW  System 
Installed 


ISTF,  OAR 
(Testing  must 
place  the 
platform 
functions,  all 
sensors,  and 
all  platform 
operators  in 
the  same 
environment 
at  the  same 
time.) 


ADS  Benefits 


Limited  to  Significant. 

-  Limited  benefit  if  all  required 
stimulators  are  local  and  can 
present  the  same  environment  at 
the  same  time.  * 

-  If  all  stimulators  are  not  local, 
testing  may  be  accomplished  by 
testing  the  system  sequentially 
in  different  facilities. 

-  ADS  can  provide  a  common 
test  environment  to  all  sensors 
simultaneously  (within  the 
latency  capability  of  the 
architecture). 

-  ADS  may  be  less  expensive 
than  facility  upgrades. 

-  ADS  may  save  schedule  over 

sequential  testing. _ 

Limited  to  Significant. 

-  Limited  benefit  if  all  required 
stimulators,  operator  interfaces, 
and  platform  functions  are  local 
and  can  present  the  same 
environment  at  the  same  time. 

-  If  all  stimulators,  operator 
interfaces,  and  platform 
functions  are  net  local,  testing 
may  be  accomplished  by  testing 
the  system  sequentially  in 
different  facilities.  However, 
this  limits  testing  to  nonreactive 
scenarios. 

-  ADS  can  provide  a  common 
test  environment  to  all  sensors, 
operators,  and  platform 
functions  simultaneously  (within 
the  latency  capability  of  the 
architecture). 

-  ADS  allows  reactive  test 
scenarios  (within  the  latency 
capability  of  the  architecture). 

-  ADS  may  be  less  expensive 
than  facility  upgrades. 

-  ADS  may  save  schedule  over 
sequential  testing. 


Utility  Assessment 


Benefits  should  overcome  cost 
of  ADS  architecture  when  all 
stimulators  are  not  contained  in 
a  single  facility. 


ADS  has  significant  benefits  if  a 
single  facility  can’t  create  the 
needed  environment. 
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Test 

Objective 

Type 

EW 

System 

Type 

Current  DoD 
Test  Process 
Facility 

ADS  Benefits 

Utility  Assessment 

Closed  Loop 

Single 

Device 

DSM 

None 

Moderate. 

-  ADS  provides  the  potential  for 
reactive  manned  threats  to 
interact  with  the  DSM.  Reactive 
manned  threats.allow  system 
effectiveness  to  be  evaluated. 

-  Real-time  DSMs  may  allow 
dissimilar  alternatives  to  be 
directly  compared  during 
analysis  of  alternatives  phase  of 
development. 

Benefits  may  offset  the  cost  of 
ADS  architecture. 

Single 

Device 

Uninstalled 

SIL,  HITL 

Moderate. 

-  Local  stimulators  should  have 
the  ability  to  generate  the 
correct  environment  or  the  SUT 
is  untestable  at  that  location. 

-  ADS  adds  the  potential  ability 
to  link  hardware  installed  in  one 
facility  to  interact  with  manned 
threat  simulators  for 
effectiveness  testing  and/or  with 
human  platform  operators )  to 
allow  more  reactive  and 
operationally  realistic  testing. 

-  ADS  may  facilitate  closed- 
loop  effectiveness  testing  before 
hardware  is  in  an  easily 
transported  configuration. 

-  ADS  potentially  reduces  time 
to  correct  problems  by  leaving 
the  SUT  in  its  development 
environment. 

Benefits  may  offset  the  cost  of 
ADS  architecture. 
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Single  ISTF,  OAR  Moderate. 

Device  -  Local  stimulators  should  have 

Installed  the  ability  to  generate  the 

correct  environment  or  the  SUT 
is  untestable  at  that  location. 

-  ADS  adds  the  potential  ability 
to  link  hardware  installed  in  one 
facility  to  interact  with  manned 
threat  simulators  for 
effectiveness  testing  and/or  with 
human  platform  operator(s)  to 
allow  more  reactive  and 
operationally  realistic  testing. 

-  ADS  may  allow  concurrent 
electromagnetic 
compatibility/interference 
testing  and  effectiveness  testing 
reducing  the  risk  of  executing 
sequential  tests. 

-  ADS  potentially  allows 
training  assets  to  be  used  to 
increase  signal  density. 

-ADS  potentially  allows  OAR 
emitter  only  threats  to  use 
closed-loop  representations  of 
the  same  threat  at  other  facilities 
to  increase  closed-loop  threat 
density. 

-  ADS  potentially  allows  real¬ 

time,  high  fidelity  flyout  of 
semi-active  missiles  with  OAR 
track  loop  errors. _ 


Integrated  None  Moderate. 

EW  System  -  ADS  provides  the  potential  for 

DSM  reactive  manned  threats  to 

interact  with  the  DSM.  Reactive 
manned  threats  allow  system 
effectiveness  to  be  evaluated. 

-  Real-time  DSMs  may  allow 
dissimilar  alternatives  to  be 
directly  compared  during 
analysis  of  alternatives  phase  of 
development. 


Utility  Assessment 


Benefits  may  offset  the  cost  of 
ADS  architecture. 


Benefits  may  offset  the  cost  of 
ADS  architecture. 
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Test 

Objective 

Type 

EW 

System 

Type 

Current  DoD 
Test  Process 
Facility 

ADS  Benefits 

Utility  Assessment 

Integrated 
EW  System 
Uninstalled 

SIL,  HITL 
(Testing  must 
place  all 
sensors  in  the 

same 

environment 
as  manned 
threat 
simulators 
and  all  must 
interact.) 

Limited  to  Significant. 

-  Limited  benefit  if  all  required 
stimulators  are  local,  can  present 
the  same  environment  at  the 
same  time,  and  include  manned 
threat  simulators. 

-  If  all  stimulators  and  manned 
threat  simulators  are  not  local, 
testing  may  be  accomplished  by 
testing  the  system  sequentially 
in  different  facilities,  however 
risk  increases  and  sequential 
testing  significantly  limits  the 
reactive  elements  in  the 
scenario. 

-  ADS  can  provide  a  common 
test  environment  to  all  sensors 
simultaneously  and  manned 
threat  simulators  (within  the 
latency  capability  of  the 
architecture). 

-  ADS  may  be  less  expensive 
than  facility  upgrades. 

-  ADS  may  save  schedule  over 
sequential  testing. 

-  ADS  may  add  significant 
realism  to  the  scenario. 

-  ADS  may  facilitate  closed- 
loop  effectiveness  testing  before 
hardware  is  in  an  easily 
transported  configuration. 

-  ADS  potentially  reduces  time 
to  correct  problems  by  leaving 
the  SUT  in  its  development 
environment. 

ADS  has  significant  benefits  if  a 
single  facility  can’t  create  the 
needed  environment. 

25 


H  Q  H 


EW 

System 

Type 

Current  DoD 
Test  Process 
Facility 

ADS  Benefits 

Utility  Assessment 

Integrated 
EW  System 
Installed 

ISTF,  OAR 
(Testing  must 
place  the 
platform 
functions,  all 
sensors, 

manned  threat 
simulators 
and  all 
platform 
operators  in 
the  same 
environment 
at  the  same 
time.) 

Limited  to  Significant. 

-  Limited  benefit  if  all  required 
stimulators,  operator  interfaces, 
and  platform  functions  are  local 
and  can  present  the  same 
environment  at  the  same  time. 

-  If  all  stimulators,  manned 
threat  simulators  and  platform 
operators  are  not  local,  testing 
may  be  accomplished  by  testing 
the  system  sequentially  in 
different  facilities,  however  risk 
increases  and  sequential  testing 
significantly  limits  the  reactive 
elements  in  the  scenario. 

-  ADS  can  provide  a  common 
test  environment  to  all  sensors, 
operators,  and  platform 
functions  simultaneously  (within 
the  latency  capability  of  the 
architecture). 

-  ADS  allows  reactive  test 
scenarios  (within  the  latency 
capability  of  the  architecture). 

-  ADS  may  be  less  expensive 
than  facility  upgrades. 

-  ADS  may  save  schedule  over 
sequential  testing. 

-  ADS  may  allow  concurrent 
electromagnetic 
compatibility/interference 
testing  and  effectiveness  testing 
reducing  the  risk  of  executing 
sequential  tests. 

-  ADS  potentially  allows 
training  assets  to  be  used  to 
increase  signal  density. 

-ADS  potentially  allows  OAR 
emitter  only  threats  to  use 
closed-loop  representations  of 
the  same  threat  at  other  facilities 
to  increase  closed-loop  threat 
density. 

-  ADS  potentially  allows  real¬ 
time,  high  fidelity  flyout  of 
semi-active  missiles  with  OAR 
track  loop  errors. 

ADS  has  significant  benefits  if  a 
single  facility  can’t  create  the 
needed  environment. 

Table  1  was  limited  to  EW  devices  and  integrated  system  testing  which  cover  testing  one  on  one 
and  one  on  few.  EW  devices  are  not  limited  to  these  types  of  engagements  since  the  platforms 
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they  support  are  not  generally  deployed  in  single-ship  formations.  The  effect  on  mission  success 
of  EW  devices  in  operationally  relevant  multi-ship  formations  is  equally  important  to  understand. 
Likewise,  providing  the  multi-ship  formation  with  the  electronic  support  measures  (ESM) 
expected  in  combat  is  essential  if  the  developer,  tester,  or  user  wants  to  fully  understand  the 
combined  impact  of  all  the  EW  systems  involved  in  mission  success.  This  is  generally  described 
as  systems  of  systems  testing. 

ADS  has  utility  for  systems  of  systems  testing  since* it  provides  affordable  access  to  high  fidelity 
manned  simulations  of  additional  players.  (Reference  3)  Traditional  methods  of  evaluating 
systems  of  systems  used  either  model-on-model  evaluations  or  live  player  through  exploitation  of 
exercises  or  actual  operations.  All  these  approaches  are  limited.  While  simulators  and  linking 
technology  are  available  today,  they  remain  the  limiting  factor  in  systems  of  systems  T&E.  As 
initiatives  such  as  Simulation  Based  Acquisition  and  HLA  begin  to  bear  fruit,  the  tester  should 
have  more  raw  materials  available  to  create  rich  synthetic  environments  for  ADS-based  systems 
of  systems  T&E. 
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5.0  Summary 


The  findings  from  the  JADS  SPJ  test  coupled  with  the  experience  gleaned  from  the  HLA  EPF, 
TSLA,  and  ADEWS  were  extrapolated  to  EW  systems  other  than  the  airborne  SPJ.  Distributed 
interactive  simulations  are  already  being  used  to  test  EW  systems.  Geographical  distribution  is  a 
logical  step  in  the  evolution  of  EW  testing,  and  several  EW  test  facilities  are  experimenting  in 
this  area.  ADS  testing  is  not  addressed  in  the  DoD'EW  test  process  description;  however,  ADS 
can  be  useful  to  EW  testing.  It  may  provide  a  method  for  improving  the  validation  of  DSMs.  It 
provides  the  opportunity  to  avoid  sequential  testing  by  combining  the  capabilities  of  different 
facilities.  It  provides  an  alternative  to  improving  a  facility  to  create  a  single  environment  for 
adequate  testing  of  integrated  EW  systems.  It  may  be  the  only  way  to  affordably  test  systems  of 
systems. 

A  more  definitive  statement  on  utility  is  not  possible  because  of  the  inability  to  clearly  identify 
cost  savings  and  because  the  benefits  of  using  ADS  vary  according  to  the  type  of  system  being 
tested.  In  general,  the  cost  of  this  architecture  is  more  likely  to  be  recovered  over  the  entire 
development  cycle  than  it  is  in  a  single  test  event.  All  types  of  EW  systems  can  be  tested  in 
some  fashion  using  ADS.  However,  stand-alone  EW  functions  are  less  likely  than  integrated 
EW  systems  to  find  the  benefits  worth  the  costs.  The  largest  barrier  to  ADS  testing  in  EW  is  the 
availability  of  linkable  EW  test  infrastructure.  However,  the  missing  infrastructure  already 
limits  the  amounts  and  types  of  traditional  EW  testing. 

This  report  provides  a  general  assessment  of  the  usefulness  of  ADS  testing  for  different  EW 
system  functions.  Detailed  requirements  were  not  developed,  however  the  TSLA  study  provides 
more  information  on  message  sizes  and  rates.  Also,  some  characteristics  of  specific  EW  systems 
may  have  been  overlooked,  which  could  impact  the  ADS  design  and  feasibility  of 
implementation.  The  extrapolation  of  JADS  EW  Test  results  to  EW  system  functions  was  based 
on  informed  conjecture  without  rigorous  analysis  or  supporting  data.  Applications  were  assumed 
to  be  feasible  unless  there  was  evidence  to  the  contrary. 
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Appendix  A  -  List  of  Acronyms 


ACETEF 

ADEWS 

ADS 

AFEWES 

AFMAN 

AFOTEC 

ALQ-131 


ARPA 

ATEWES 

C4I 

C4ISR 

COMM 

COTS 

CREWS 

CROSSBOW 

DIS 

DMSO 

DoD 

DSI 

DSM 

DT&E 

ECM 

EMC 

EMI 

EO 

EPF 

ES 

ESM 

ETE 

EW 

FEDEP 

GPS 

HITL 

HLA 

IADS 

ICD 


Air  Combat  Environment  Test  and  Evaluation  Facility,  Patuxent  River, 
Maryland;  Navy  facility 

Advanced  Distributed  Electronic  Warfare  System;  Army  sponsored 
advanced  distributed  simulation 

Air  Force  Electronic  Warfare  Evaluation  Simulator,  Fort  Worth,  Texas 
Air  Force  manual 

Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force  Base, 
New  Mexico 

a  mature  self-protection  jammer  system;  an  electronic  countermeasures 
system  with  reprogrammable  processor  developed  by  Georgia  Tech  Research 
Institute 

Advanced  Research  Projects  Agency 

Advanced  Tactical  Electronic  Warfare  Environment  Simulator 

command,  control,  communications,  computers  and  intelligence 

command,  control,  communications,  computers,  intelligence,  surveillance 

and  reconnaissance 

communications 

commercial  off-the-shelf 

Covert  Remote  Electronic  Warfare  System 

Office  of  the  Secretary  of  Defense  committee  under  the  Director,  Test 
Systems  Engineering  and  Evaluation 
distributed  interactive  simulation 

Defense  Modeling  and  Simulation  Organization,  Alexandria,  Virginia 

Department  of  Defense 

Defense  Simulation  Internet 

digital  system  model 

developmental  test  and  evaluation 

electronic  countermeasures 

electromagnetic  compatibility 

electromagnetic  interference 

electro-optical 

engineering  protofederation 

entity  state;  electronic  support 

electronic  support  measures 

JADS  End-to-End  Test 

electronic  warfare 

federation  development  and  execution  process 

global  positioning  system 

hardware-in-the-loop 

high  level  architecture 

Integrated  Air  Defense  System 

interface  control  document 
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IEEE 

IP 

IR 

ISTF 

JADS 

JETS 

JMASS 

Joint  STARS 

JT&E 

JTF 

MIL 

MITRE 

MOE 

MOP 

OAR 

OMDT 

OSD 

OT&E 

RCS 

RF 

RTI 

SAR 

SIL 

SIMNET 

SIT 

SPJ 

SUT 

SWEG 

T&E 

T-l 


TCP 

TSLA 

TSPI 

UDP 

v&v 

VV&A 


Institute  of  Electrical  and  Electronics  Engineers 

initial  point;  Internet  protocol 

infrared 

installed  systems  test  facility 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

JammEr  Techniques  Simulator 

Joint  Modeling  and  Simulation  System 

Joint  Surveillance  Target  Attack  Radar  System 

joint  test  and  evaluation 

joint  test  force 

military 

company  that  provides  engineering  services 
measures  of  effectiveness 
measures  of  performance 
open  air  range 

object  model  development  tool 
Office  of  the  Secretary  of  Defense 
operational  test  and  evaluation 
radar  cross  section 
radio  frequency 
runtime  infrastructure 
synthetic  aperture  radar 

system  integration  laboratory;  system-in-the-loop 

simulator  network 

JADS  System  Integration  Test 

self-protection  jammer 

system  under  test 

Simulated  Warfare  Environment  Generator  at  Air  Combat  Environment  Test 
and  Evaluation  Facility,  Patuxent  River,  Maryland 
test  and  evaluation 

digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1-544  megabits 
per  second 

transmission  control  protocol 
Threat  Simulator  Linking  Activity 
time-space-position  information 
user  datagram  protocol 
verification  and  validation 
verification,  validation  and  accreditation 
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Appendix  B  -  Kaminski  Memorandum 
Directing  Implementation  of  HLA  for  DoD  Simulations 


Under  Secretary  of  Defense 
(Acquisition  and  Technology) 


Sept.  10,  1596 

MEMORANDUM  SECRETARIES  OF  THE  MILITARY  DEPARTMENTS 
POR.  CHAIRMAN  OF  THE  JOINT  CHIEFS  OF  STAFF 

UNDER  SECRETARIES  OF  DEFENSE 
ASSISTANT  SECRETARIES  OF  DEFENSE 
GENERAL  COUNCIL  OF  THE  DEPARTMENT  OF  DEFENSE 
INSPECTOR  GENERAL  OF  THE  DEPARTMENT  OF  DEFENSE 
DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 
ASSISTANTS  TO  THE  SECRETARY  OF  DEFENSE 
DIRECTOR  OF  ADMINISTRATION  AND  MANAGEMENT 
DIRECTORS  OF  THE  DEFENSE  AGENCIES 


SUBJECT:  DoD  High  Level  Architecture  (HLA)  for  Simulations 

References:  a.  DoD  Directive  5000.59,  "DoD  Modeling  and  Simulation 

(M&S)  Management,"  January  4,  1994 

b.  DoD  5000. 59-P,  "DoD  Modeling  and  Simulation  Master 

Plan  (MSMP),"  October  1995 


Under  the  authority  of  reference  (a),  and  as  prescribed  by  reference  (b),  I  designate  the  High 
Level  Architecture  as  the  standard  technical  architecture  for  all  DoD  simulations. 

The  baseline  HLA  is  defined  by  three  inter-related  elements:  HLA  Rules  Version  1.0  (v.1.0), 
HLA  Interface  Specification  v.1.0,  and  HLA  Object  Model  Template  v.1.0.  The  evolution  of 
the  HLA  will  be  managed  by  the  DoD  Executive  Council  for  Modeling  and  Simulation 
(EXCIMS)  through  its  Architecture  Management  Group  (AMG).  This  structure  provides  a 
means  for  the  DoD  Components  to  identify  and  address  any  emergent  issues  in  subsequent 
refinements  to  the  HLA.  Compliance  with  the  HLA  does  not  mandate  the  use  of  any  particular 
implementation  of  supporting  software  such  as  the  Runtime  Infrastructure. 

DoD  Components  shall  review  all  of  their  simulation  projects  and  programs  by  the  second 
quarter  fiscal  year  (FY)  1997  in  order  to  establish  plans  for  near-term  compliance  with  the 
HLA.  The  Department  shall  cease  further  development  or  modification  of  all  simulations  which 
have  not  achieved,  or  are  not  in  the  process  of  achieving,  HLA-compliance  by  the  first  day  of 
FY  1999,  and  shall  retire  any  non-compliant  simulations  by  the  first  day  of  FY  2001 .  EXCIMS 
is  to  monitor  progress  and  advise  me  if  any  emergent  events  affect  their  viability. 

To  monitor  compliance  with  the  HLA,  the  DoD  Components  shall  submit  an  initial  report  to 
the  Defense  Modeling  and  Simulation  Office  (DMSO)  by  June  30,  1997,  which  summarizes 
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their  HLA-compliance  intentions  for  each  simulation  the  Component  owns  or  sponsors, 
organized  into  three  categories: 


•  HLA-compliance  actions  initiated  immediately 

•  HLA-compliance  actions  initiated  at  a  specified  future  date 

•  no  HLA  compliance  planned  (thus  requiring  eventual  retirement  or  a  waiver) 

The  DoD  Components  shall  submit  periodic  updates  to  these  initial  reports  as  required  to 
ensure  their  accuracy  and  completeness.  DMSO  shall  establish  a  mechanism  to  provide  for 
formal  certification  of  compliance  and  shall  provide  me  with  periodic  reports  on  the 
Department's  progress  towards  compliance  with  the  HLA. 

If  a  Component  believes  it  is  impractical  for  a  simulation  to  comply  with  the  HLA,  or  that 
HLA-compliance  cannot  be  achieved  in  a  timely  manner,  it  may  submit  a  waiver  request  to  the 
Director  of  Defense  Research  and  Engineering,  the  Chair  of  the  EXCIMS.  In  consultation  with 
the  EXCIMS  and  its  Training,  Analysis,  and  Acquisition  Councils,  I  will  then  decide  if  an 
exception  to  the  HLA-compliance  requirement  is  warranted,  and  if  so,  the  form  of  that 
exception. 

This  mandate  for  HLA-compliance  supersedes  all  previous  requirements  for  DoD  simulations 
to  comply  with  other  simulation  standards  such  as  Distributed  Interactive  Simulation  or 
Aggregate-Level  Simulation  Protocol.  It  is  expected  that  new  industry  standards  to  support  the 
HLA  will  emerge.  In  consultation  with  the  EXCIMS  and  its  AMG,  I  will  evaluate  the  suitability 
of  such  standards  for  the  Department  as  they  are  established. 

The  DoD  point  of  contact  for  the  HLA  is  the  Defense  Modeling  and  Simulation  Office  at  (703) 
998-0660  or  hla@dmso.mil.  The  HLA  documents  are  available  at  http://www.dmso.mil/. 

\original  signed) 

Paul  G.  Kaminski 
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Appendix  C  -  Simple  ADS  Concept  Model 


The  following  discussion  presents  a  general  concept  model  intended  to  provide  those  unfamiliar 
with  Joint  Advanced  Distributed  Simulation  (JADS)  with  a  simple  frame  of  reference  for 
understanding  where  distributed  testing  can  be  inserted  for  enhancing  test  and  evaluation  (T&E) 
environments.  Figure  Cl  illustrates  the  simplest  representation  of  a  weapon  system  or  electronic 
warfare  (EW)  device.  The  sensor  provides  a  representation  of  the  outside  world  or  environment 
to  the  processing  function.  The  sensor  may  require  some  form  of  transmitted  energy  (the 
flashlight  in  the  figure)  to  illuminate  the  environment  for  the  sensor.  The  processor  acts  on  the 
information  provided  by  the  sensor  and,  if  appropriate,  takes  some  action  that  ultimately  changes 
the  environment. 

The  sensor  can  be  an  actual  radio  frequency  (RF)  sensor  (as  in  the  case  of  a  radar  warning 
receiver,  an  intelligence  receiver,  a  data  link,  etc.),  a  non-RF  sensor  (ultraviolet,  infrared,  a  laser 
detector,  etc.),  or  it  may  simply  be  the  interface  to  the  platform’s  internal  communications  bus  as 
in  the  case  of  a  simple  expendables  dispenser.  Illumination  for  the  sensor  can  be  RF,  electro- 
optical  (OE)/infrared  (IR),  coherent  or  incoherent.  However,  the  illuminator  is  controlled  by  the 
processor.  Its  contribution  to  the  environment  would  not  be  expected  to  be  present  unless  the 
device  is  present.  The  sensor  may  or  may  not  require  an  illuminator.  The  processor  can  be 
hardware,  human  or  both  depending  on  how  the  test  designer  chooses  to  model  the  system.  It  is  a 
mechanism  that  maps  sensor  information  into  some  response.  Responses  are  quite  varied. 
Responses  include  cueing  the  operator,  automatic  course  change,  a  track  file  hand-off  to  another 
onboard  system,  information  hand-off  to  another  platform,  or  a  directed  energy  attack  on  the 
threat. 


Processing  Response  Feedback 


Active  Illuminator  (optional) 


Figure  Cl.  Simple  System 

Distributed  simulation  can  be  introduced  into  the  system  in  several  ways.  First  simulation  can  be 
used  to  supply  the  entire  environment  the  sensor  sees.  A  stimulator  takes  inputs  from  the 
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simulation  architecture  and  recreates  it  in  a  format  the  sensor  can  “see.”  This  approach  was  used 
in  the  System  Integration  Test  (SIT)  where  the  environment  seen  by  the  missile  was  controlled  by 
other  players  in  the  architecture.  Next,  simulation  can  be  used  to  add  information  to  a  scene 
being  observed  by  the  sensor.  Think  of  this  as  breaking  into  the  communication  path  between 
the  sensor  and  the  brain  to  add  features  to  the  environment  that  are  not  really  there.  JADS  used 
this  approach  in  the  End-to-End  (ETE)  Test  where  more  than  10,000  entities  were  added  to  the 
synthetic  aperture  radar  (SAR)  scene.  Finally,  simulation  can  be  used  to  insert  the  response  into 
the  environment  being  seen  by  another  sensor.  This  was  done  in  the  EW  Test  where  the  jammer 
responses  were  injected  into  the  RF  environment  seen  by  the  threat  systems.  All  three  methods 
can  be  used  in  EW  testing.  This  is  shown  in  Figure  C2. 

It  is  important  to  note  from  the  figure,  the  number  of  different  possible  representations  of  the 
sensor,  processor,  illuminator,  and  response.  In  distributed  testing,  each  of  these  need  not  be  the 
same  maturity  in  order  to  be  connected.  For  example,  a  military  hardware  processor  can  be 
connected  to  a  digital  simulation  of  the  sensor.  T&E  will  likely  use  the  representation  that  makes 
sense  for  the  test  objective  desired.  This  means  that  each  of  the  communications  paths  will  be 
built  based  on  the  representation  of  the  two  items  the  path  is  connecting.  In  the  military 
hardware,  digital  simulation  example  above,  the  designer  may  use  the  military  standard  data  bus 
that  would  normally  connect  the  sensor  and  processor,  or  the  designer  may  elect  to  use 
commercial  computer  communications  hardware  and  protocols  (e.g.,  Ethernet  and  transmission 
control  protocol  [TCP]/Internet  protocol  [IP])  and  place  an  interface  unit  in  front  of  the  military 
hardware. 


Actual 

Processor 


Bread  Board 

Simulation 


MIL  Hardware 

Simulation 


Live 

Simulation 

Construction 


MIL  Hardware 

Simulation 


COMM  =  communications 


COTS  =  commercial  off-the-shelf 


MIL  =  military 


Figure  C2.  System  Connection  Points  to  Synthetic  Environment 


Figure  C2  illustrates  the  different  ways  each  player  in  an  distributed  interactive  simulation  might 
be  linked  to  other  players.  Multiple  players  will  need  to  be  connected  to  create  a  test 
architecture.  The  challenge  to  the  test  designer  is  to  step  past  the  traditional  hardware  boundaries 
and  divide  the  problem  into  functions  that  can  be  combined  to  realistically  represent  the  players. 
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Distributed  test  facilities  already  address  EW  problems  in  this  fashion. 

The  next  challenge,  once  the  functions  are  defined,  is  to  find  a  facility  or  simulation  that  has  the 
function(s)  required.  This  is  illustrated  in  the  EW  Test  in  several  ways.  The  “aircraft”  was 
composed  of  a  radar  cross  section  (RCS)  database  at  the  Air  Force  Electronic  Warfare  Evaluation 
Simulator  (AFEWES)  combined  with  a  time-space-position  information  (TSPI)  script  played  into 
the  architecture  from  JADS.  The  Phase  3  jammer  had  two  transmitter  sets.  The  first  was  the 
pod’s  internal  military  hardware  which  radiated  into  the  anechoic  chamber  at  Air  Combat 
Environment  Test  and  Evaluation  Facility  (ACETEF)  to  simulate  electromagnetic  capability 
(EMC)/electromagnetic  interference  (EMI)  testing.  The  second  was  a  laboratory  jammer 
technique  generator/simulator,  the  JammEr  Technique  Simulator  (JETS)  at  AFEWES.  The  JETS 
jamming  waveforms  were  injected  into  the  RF  environment  that  the  AFEWES  threat  simulators 
used.  Similarly,  the  AFEWES  threat  simulators  had  two  transmitters.  The  first  was  at  AFEWES 
and  it  was  used  to  “illuminate”  the  synthetic  aircraft  (RCS  and  TSPI)  for  the  environment  at 
AFEWES.  The  second  was  the  Advanced  Tactical  Electronic  Warfare  Environment  Simulator 
(ATEWS)  equipment  at  ACETEF.  This  hardware  generated  the  RF  energy  that  was  injected  into 
the  pod  behind  the  antenna  to  create  the  environment  for  the  pod. 

It  is  important  to  realize  that  the  environment  at  AFEWES  was  not  the  same  as  the  environment 
seen  by  the  pod.  Latency  differences  placed  the  aircraft  at  slightly  different  locations  in  the  two 
environments.  JADS  overcame  this  by  taking  all  miss  distance  data  from  the  AFEWES 
environment.  There  were  other  obvious  differences.  The  aircraft  fire  control  radar  was  not 
recreated  in  the  AFEWES  environment.  During  excursion  runs,  ACETEF  used  the  Synthetic 
Warfare  Environment  Generator  model  (SWEG)  to  add  two  additional  threats  into  the  pod’s 
environment.  These  were  not  recreated  in  the  AFEWES  environment  either  because  of  schedule 
constraints.  The  decision  to  add  the  extra  threats  was  made  one  week  before  our 
verification/validation  testing  of  the  ACETEF  federate  and  the  architecture  certification.  The 
final  difference  was  in  the  waveforms  at  each  facility.  Because  the  RF  waveform  was  not 
digitized  and  transmitted,  it  was  possible  that  there  were  waveform  differences  between  the  two 
facilities.  Additional  instrumentation  was  needed  to  record  the  actual  RF  environment  at  each 
facility  during  test,  and  the  waveforms  at  each  facility  had  to  be  examined  closely  and  validated 
prior  to  validating  the  architecture. 

Since  the  architecture  did  not  transmit  digitized  waveforms,  JADS  had  to  find  a  way  to 
communicate  threat  and  pod  state  changes.  JADS  created  simple  brief  messages  to  indicate  state 
changes  in  the  threat  or  the  pod.  Only  these  brief  mode  descriptions  were  transmitted  between 
AFEWES  and  ACETEF.  The  actual  RF  was  created  at  each  site  based  on  a  common 
understanding  of  the  mode  descriptions.  This  is  illustrated  in  Figure  C3. 
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Figure  C3.  EW  Test  Architecture 

This  discussion  is  intended  to  start  the  test  designer  down  the  path  of  creating  a  distributed  test. 
There  are  other  tools  needed  to  fully  create  and  describe  a  useful  test  design  methodology.  The 
high  level  architecture  (HLA)  community  is  developing  a  set  of  processes  and  tools  that  will  help 
test  designers  create  distributed  test  architectures.  These  processes  and  tools  need  to  be  added  to 
the  organization’s  program  management  and  system  engineering  processes  to  manage  the  risks 
involved  in  creating  distributed  simulation  and  distributed  test  architectures.  The  Federation 
Development  Execution  Process  (FEDEP)  describes  the  steps  that  designers  must  go  through  to 
create  distributed  simulation  and  distributed  test  environments.  Several  overlays  such  as  security 
and  verification,  validation,  and  accreditation  (VV&A)  are  being  created  to  connect  these 
processes  to  the  FEDEP.  The  tools  are  directed  primarily  at  HLA  applications.  These  tools 
include  the  object  model  development  tool  (OMDT)  (which  describes  the  message  formats,  who 
creates  the  message,  and  who  uses  the  message)  and  the  Federation  Execution  Planners 
Workbook  (which  ties  the  messages  to  hardware  within  the  architecture  and  allows  issues  such  as 
latency  and  bandwidth  to  be  addressed).  These  tools  coupled  with  traditional  systems  and 
software  engineering  tools  (such  as  interface  control  documents),  system  specifications,  software 
requirement  specifications,  and  configuration  management  tools)  will  help  the  designer  build  and 
execute  a  distributed  test. 
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